Papers with language model pre-training
Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts (2022.aacl-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a longstanding NLP task that consists of identifying an entity in a sentence or document. |
| Approach: | They construct a dataset of seven entity types annotated over 11,382 tweets . they provide a set of language model baselines and analyze the performance of the model . |
| Outcome: | The proposed dataset contains seven entity types annotated over 11,382 tweets . the authors focus on short-term degradation of NER models over time and strategies to fine-tune a language model over different periods . |
Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval (2023.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models have been successful in knowledge-intensive tasks, but recent research calls into question the robustness of these singlevector models. |
| Approach: | They propose to exploit knowledge in a pre-trained language model for dense passage retrieval by aggregating contextualized token embeddings into a dense vector. |
| Outcome: | The proposed model significantly improves the effectiveness of dense retrieval models on in-domain and zero-shot evaluations without introducing substantial training overhead. |
Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval (2024.acl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that Transformer-based language models lose information in the middle of input sequences, especially in the context of web document retrieval. |
| Approach: | They examine position biases at multiple stages of the training pipeline for an encoder-decoder neural retrieval model, namely language model pre-training, contrastive pre- training, and contrastive fine-tuning. |
| Outcome: | The proposed model generates embeddings that better capture the beginning of the input content, with fine-tuning further aggravating this effect. |
Enhancing Clinical BERT Embedding using a Biomedical Knowledge Base (2020.coling-main)
Copied to clipboard
| Challenge: | Domain knowledge is important for building Natural Language Processing (NLP) systems for low-resource settings, such as in the clinical domain. |
| Approach: | They propose a joint method for adding knowledge base information from the Unified Medical Language System (UMLS) into language model pre-training for some clinical domain corpus. |
| Outcome: | The proposed method outperforms existing models on three clinical domain tasks with no knowledge base information. |
Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn general representations of text can achieve sub-optimal performance in low-resource scenarios. |
| Approach: | They propose to use language model pre-training and multi-task learning to learn robust representations but these methods can achieve sub-optimal performance in low-resource scenarios. |
| Outcome: | The proposed model outperforms strong baselines on the GLUE benchmark and can be adapted to new tasks efficiently and effectively. |
Dict-BERT: Enhancing Language Model Pre-training with Dictionary (2022.findings-acl)
Copied to clipboard
Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, Meng Jiang
| Challenge: | Pre-trained language models (PLMs) capture word semantics in different contexts, hence the embeddings of rare words on the tail are poorly optimized. |
| Approach: | They propose to leverage definitions of rare words in dictionaries to enhance language model pre-training by leveraging dictionary definitions. |
| Outcome: | The proposed model improves understanding of rare words and boosts performance on various NLP downstream tasks. |
DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI (2024.findings-eacl)
Copied to clipboard
Jianguo Zhang, Kun Qian, Zhiwei Liu, Shelby Heinecke, Rui Meng, Ye Liu, Zhou Yu, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | DialogStudio is the largest and most diverse collection of dialogue datasets . existing datasets lack diversity and comprehensiveness, authors say . |
| Approach: | They introduce DialogStudio: the largest and most diverse collection of dialogue datasets . DialogStuio aggregates more than 80 diverse dialogue dataset . |
| Outcome: | a new dataset is created to improve the quality and diversity of dialogue datasets . DialogStudio is the largest and most diverse collection of dialogue data . |
A Neural Local Coherence Analysis Model for Clarity Text Scoring (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for scoring text clarity use local coherence between adjacent sentences . local cohesion is one of the main properties to identify whether a text is well-structured or not. |
| Approach: | They propose a method for scoring text clarity by utilizing local coherence between adjacent sentences. |
| Outcome: | The proposed method improves on the PeerRead benchmark dataset. |
Blockwise Self-Attention for Long Document Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in pre-training and fine-tuning methods have drastically reshaped the landscape of natural language processing research. |
| Approach: | They propose a lightweight BERT model that introduces sparse block structures into the attention matrix to reduce memory consumption and training/inference time. |
| Outcome: | The proposed model uses 18.7-36.1% less memory and 12.0-25.1% more time to learn compared to an advanced BERT-based model, RoBERTa. |
On the Role of Bidirectionality in Language Model Pre-Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on language model pre-training explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult. |
| Approach: | They propose a framework that generalizes prior approaches to pre-training language models by focusing on bidirectionality and controlling each of them separately. |
| Outcome: | The proposed framework generalizes prior approaches including fully unidirectional models like GPT, fully bidirectional models and hybrid models like CM3 and prefix LM. |
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that convolutions have been successful in natural language learning. |
| Approach: | They propose a convolutional approach to construct relative position embeddings in self-attention layers and propose 'compact attention' they propose multiple ways to integrate convolutions into Transformer self- attention. |
| Outcome: | The proposed composite attention improves performance on multiple downstream tasks, replacing absolute position embeddings, and is more expressive than convolutions in NLP. |
Learning Spoken Language Representations with Neural Lattice Language Modeling (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for pretraining language models do not consider spoken language properties. |
| Approach: | They propose a framework that trains neural lattice language models to provide contextualized representations for spoken language understanding tasks. |
| Outcome: | The proposed framework outperforms baselines on spoken inputs on intent detection and dialogue act recognition datasets. |
Phone-ing it in: Towards Flexible Multi-Modal Language Model Training by Phonetic Representations of Data (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are increasingly applied in ways that are agnostic to targeted downstream tasks. |
| Approach: | They propose a multi-modal approach to train language models using whatever text and/or audio data might be available in a language. |
| Outcome: | The proposed approach improves on pre-trained models on Swahili and Kinyarwanda data, with an improvement of up to 6% over models that are trained from scratch. |
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows gains from pre-training and fine-tuning that are multi-task . but it can be difficult to know which intermediate tasks will best transfer . |
| Approach: | They propose a large-scale learning stage for pre-finetuning between pre-training and fine-tun. |
| Outcome: | The proposed model improves performance on pretrained discriminators and generation models on a wide range of tasks while improving sample efficiency during fine-tuning. |
Energy and Carbon Considerations of Fine-Tuning BERT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work quantifying energy costs and associated carbon emissions has focused on pretraining and fine-tuning. |
| Approach: | They perform an empirical study to quantify the energy requirements of language model fine-tuning in the context of pretraining and inference. |
| Outcome: | The proposed model fine-tuning energy and carbon footprints are compared with pre-training and inference energy requirements and outline recommendations for NLP researchers and practitioners. |
Instance Regularization for Discriminative Language Model Pre-training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have optimized independent strategies of ennoising or denosing . Existing methods treat training instances equally throughout the training process . |
| Approach: | They propose to use ennoising and denoising to train discriminative pre-trained language models . they propose to model the complexity of restoring the original sentences from corrupted ones . |
| Outcome: | Experimental results show that the proposed method improves pre-training efficiency, effectiveness, and robustness. |
Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to optimize language model pre-training data mixtures are difficult due to the complexity of the data mixture. |
| Approach: | They propose a method to optimize language model pre-training data mixtures by approximating cross-entropy loss via a Mixture of Data Experts (MDE). |
| Outcome: | The proposed method improves performance on a slimPajama dataset with a mixture of data experts. |